To effectively utilize the depth information from RGB-D (Red Green Blue and Depth) images and enhance the scale invariance and rotation invariance of BRISK (Binary Robust Invariant Scalable Keypoints) algorithm, an improved BRISK algorithm combined with depth information was proposed. Firstly, the keypoints were detected by the FAST (Features from Accelerated Segment Test) algorithm and their Harris corner response values were computed. Then, the entire image was divided into the same size grids, and the keypoint with the maximum Harris corner response value was reserved by each grid. Next, the scale factor of the keypoint was directly computed with the depth information of the image. Finally, the intensity centroid of the circle centered on the keypoint was calculated, and the orientation of keypoint was computed by the offset from its intensity centroid. The comparison experiment analysis of several algorithms on the scale invariance and rotation invariance was performed. The experimental results show that, compared with the BRISK algorithm, the number of correctly matched keypoints of the improved algorithm improves by more than 90% when the image's scale is changed and raises by at least 70% when the image is rotated.